Cholesky - based Methods for Sparse Least Squares : The Benefits of Regularization ∗

We study the use of black-box LDL factorizations for solving the augmented systems (KKT systems) associated with least-squares problems and barrier methods for linear programming (LP). With judicious regularization parameters, stability can be achieved for arbitrary data and arbitrary permutations of the KKT matrix. This offers improved efficiency compared to implementations based on “pure normal equations” or “pure KKT systems”. In particular, the LP matrix may be partitioned arbitrarily as (As Ad ). If AsA T s is unusually sparse, the associated “reduced KKT system” may have very sparse Cholesky factors. Similarly for least-squares problems if a large number of rows of the observation matrix have special structure. Numerical behavior is illustrated on the villainous Netlib models greenbea and pilots. 1 Background The connection between this work and Conjugate-Gradient methods lies in some properties of two CG algorithms, LSQR and CRAIG, for solving linear equations and least-squares problems of various forms. We consider the following problems: Linear equations: Ax = b (1) Minimum length: min ‖x‖ subject to Ax = b (2) Least squares: min ‖Ax− b‖ (3) Regularized least squares: min ‖Ax− b‖ + ‖δx‖ (4) Regularized min length: min ‖x‖ + ‖s‖ subject to Ax+ δs = b (5) where A is a general matrix (square or rectangular) and δ is a scalar (δ ≥ 0). LSQR [17, 18] solves the first four problems, and incidentally the fifth, using essentially the same work and storage per iteration in all cases. The iterates xk reduce ‖b − Axk‖ monotonically. CRAIG [4, 17] solves only compatible systems (1)–(2), with ‖x − xk‖ decreasing monotonically. Since CRAIG is slightly simpler and more economical than LSQR, it may sometimes be preferred for those problems. Partially supported by Department of Energy grant DE-FG03-92ER25117, National Science Foundation grant DMI-9204208, and Office of Naval Research grant N00014-90-J-1242. Systems Optimization Laboratory, Dept of Operations Research, Stanford University, Stanford, California 94305-4022 ([email protected]). Cholesky-based Methods for Least Squares 93 To extend CRAIG to incompatible systems, we have studied problem (5): a compatible system in the combined variables (x, s). If δ > 0, it is readily confirmed that problems (4) and (5) have the same solution x, and that both are solved by either the normal equations Nx = Ab, N ≡ AA+ δI, (6) or the augmented system